RESEARCH REPORT 2012-7 


CollegeBoard 


The Impact of AP® Achievement 
Institute I on Students' 

AP Performance 


By Jennifer Merriman Bausmith and Vytas Laitusis 





Jennifer Merriman Bausmith is a senior director at the College 
Board. 

Vytas Laitusis is a research scientist at the College Board. 

About the College Board 

The College Board is a mission-driven not-for-profit organization that 
connects students to college success and opportunity. Founded in 
1900, the College Board was created to expand access to higher 
education. Today, the membership association is made up of over 
6,000 of the world's leading educational institutions and is dedicated 
to promoting excellence and equity in education. Each year, the 
College Board helps more than seven million students prepare for 
a successful transition to college through programs and services in 
college readiness and college success — including the SAT® and the 
Advanced Placement Program®. The organization also serves the 
education community through research and advocacy on behalf of 
students, educators and schools. For further information, visit 
www.collegeboard.org. 

© 201 2 The College Board. College Board, Advanced Placement Program, 

AR SAT and the acorn logo are registered trademarks of the College Board. 
EXCELerator is a trademark owned by the College Board. PSAT/NMSQT is 
a registered trademark of the College Board and National Merit Scholarship 
Corporation. All other products and services may be trademarks of their 
respective owners. Visit the College Board on the Web: www.collegeboard.org. 


For more information on College 
Board research and data, visit 
www.collegeboard.org/research. 


Contents 


Executive Summary 3 

Introduction 4 

Method 4 

Data 4 

Analysis 5 

Results 5 

Descriptives 5 

Statistical Model 8 

Conclusion 9 

References 10 

Appendix 11 

Tables 

Table 1. Number of Teachers, Students, and AP® Exams in the Data File 5 

Table 2. Number of AP Exams Taken, by Students' Race/Ethnicity and Gender 6 

Table 3. Number of AP Exams Taken, by AP Subject 6 

Table 4. Mean Prior ELA Achievement on the State Assessment 7 

Table 5. Mean AP Exam Score 7 



Table 6. Mean Scale Scores 8 

Table 7. Mean AP Scores 8 

Table 8. ANCOVA Results 8 



APAI I and Student Performance 


Executive Summary 

The AP® Achievement Institute I (APAI I) is a four-day professional development program 
offered to teachers and administrators by EXCELerator™, a district reform program operated 
by the College Board. The APAI I program is designed to help teachers develop effective AP 
instructional strategies for a diverse student body and to help district, school, and curriculum 
leaders strengthen the district's infrastructure to support AP students and teachers. 

Teachers from a large urban school district on the West Coast participated in APAI I in the 
summer of 2009. The purpose of this project was to determine the impact of APAI I on 
student achievement on AP English language arts (ELA) and social studies exams. Students' 
AP Exam scores from the 2009 and 2010 administrations were examined for all participating 
teachers' students who took the AP English Language and Composition, English Literature 
and Composition, U.S. History, World History, European History, Comparative Government 
and Politics, U.S. Government and Politics, and Human Geography Exams. 

For students whose teachers participated in APAI I, achievement on AP Exams was compared 
before and after the APAI I professional development (PD). An analysis of covariance 
(ANCOVA) was conducted to compare the AP Exam scores of the participating teachers' 
students, while controlling for student prior achievement levels by year (before and after 
taking the APAI I PD), by student race/ethnicity, and by any interactions between year and 
race/ethnicity. Because students included in the file were in different grade levels in high 
school (some AP Exam takers were sophomores, some juniors, and some seniors), prior 
achievement on the state assessment was identified simply as assessment scores in the year 
prior to AP 

There were 12 teachers who received the APAI I professional development and taught one of 
the specified AP courses in both the 2008-09 and 2009-10 school years. There were slightly 
more AP students in 2009-10 compared to 2008-09 (574 and 509, respectively); students in 
2009-10 also took more AP Exams than in 2008-09 (601 and 523, respectively). Much of this 
increase was represented by Hispanic students, who took AP Exams in greater numbers. 

After the APAI I PD, more Hispanic students of lower prior achievement in absolute counts 
(though the increase was not statistically significant) were enrolled in the AP courses taught 
by the APAI teachers than before APAI I. Despite this increase, there were no statistically 
significant differences between the students' AP scores before and after the APAI I PD. In 
summary, students taking AP Exams in the second year were more diverse, representing 
lower levels of prior achievement while still managing to maintain consistent AP Exam 
performance. This suggests that the teachers were, in fact, able to provide rigorous AP 
instruction to a more diverse set of learners. 
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Introduction 

The continuing education of teachers has long been considered a cornerstone to improving 
our education system (Borko & Putnam, 1995; Darling-Hammond, 1993). The importance 
placed on continuing education or professional development can be seen in the numerous 
incarnations of professional development programs currently in operation and the billions 
of dollars spent annually by public schools to fund these activities (Guskey &Yoon, 2009; 
Zhou, 2008). Although there are numerous studies examining the efficacy of professional 
development programs in the literature, many are focused solely on the perceptions of the 
participants or employ less rigorous methodologies. In fact, in a recent review of more than 
1,300 studies, Yoon, Duncan, Lee, Scarloss, and Shapley (2007) found nine studies that met 
the minimal standards for rigor set forth by the What Works Clearinghouse. There are several 
plausible reasons why there is such a dearth of rigorous research on PD programs, including 
the time needed for real change to occur, costs, and possible lack of technical expertise. 

The current study attempts to address the desire for more rigorous research on PD through 
an examination of a pilot program for teachers and administrators called AP Achievement 
Institute I (APAI I). Instead of focusing on only teacher perceptions or correlational evidence, 
this study links student performance to their actual teacher's participation in the PD program 
while controlling for the student's prior achievement. 

The APAI I was a four-day professional development program offered to teachers and 
administrators by EXCELerator, a district reform program operated by the College Board. 

The program was designed to help teachers develop effective AP learning environments 
for a diverse student body. The APAI I was composed of two components: instructional and 
administrative. The instructional component, geared toward AP teachers, ran for four days and 
addressed topics such as defining rigor, differentiation, and scaffolding. Topics also included 
data interpretation and application, as well as the introduction of strategies that address 
reading and writing instruction. The administrative component, geared toward school-based 
administrators and curriculum leaders, ran during the first two days of the instructional 
component and included information on operating an AP program in a school as well as 
the use of data and reporting materials (see Appendix for a summary of key teacher and 
administrator objectives for the PD). 

Teachers from a large urban school district on the West Coast participated in APAI I in the 
summer of 2009. The purpose of this project was to determine the impact of APAI I on 
student achievement on AP ELA and social studies exams. AP scores from the subsequent 
2009 and 2010 administrations were examined for all participating teachers' students 
who took the AP English Language and Composition, English Literature and Composition, 

U.S. History, World History, European History, Comparative Government and Politics, U.S. 
Government and Politics, or Human Geography Exams. 


Method 

Data 

Student-level data from three years (2007-08, 2008-09, and 2009-10) were obtained from 
the participating school districts. These data included the following files: enrollment and 
demographics, course transcripts, college exam records, and state assessments. Teacher-level 
data were also obtained that contained a file on courses and sections taught. Students were 
linked to specific teacher classrooms through a unique course and section identifier. Finally, 
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a list of all teachers who participated in the APAI I professional development program in the 
summer of 2009 was obtained from the districts (n = 23). 

In order to better manage the data for analysis, two files were initially developed: a file for 
students in the 2008-09 school year and a file for students in the 2009-10 school year. To 
construct the data files for analysis, we first removed all teachers in the district teacher file 
who were not participants in the APAI I PD. Next, we deleted all teachers who did not teach 
one of the target AP courses. Next, we matched students to this remaining list of teachers 
through their course sections. We then pulled in the students' demographic data, AP Exam 
scores for each administration year, and state assessment scores on ELA from the prior 
academic year to control for prior achievement. 

Once these two files were constructed, they were concatenated to include only those 
teachers who were found in both files so that differences in their students' AP performance 
could be examined. Table 1 presents the count of teachers, students, and AP Exams in the 
final data file. The number of exams exceeds the number of students because some students 
took more than one AP Exam per year. There were 12 teachers who received the APAI I 
professional development and taught one of the specified AP courses in both the 2008-09 and 
2009-10 school years. There were slightly more AP students in 2009-10 compared to 2008-09 
(574 and 509, respectively); students in 2009-10 also took more AP Exams than those in 2008- 
09 (601 and 523, respectively). 


Table 1. 

Number of Teachers, Students, 

and AP Exams in the Data File 



2008-09 

2009-10 

AP Teachers 

12 

12 

AP Students 

509 

574 

AP Exams 

523 

601 


Analysis 

Participating teachers' student achievement on AP was compared before and after the APAI I 
professional development. Student AP performance (mean AP Exam score) from each cohort 
was the outcome of interest. An analysis of covariance (ANCOVA) was conducted to compare 
the participating teachers' students' AP Exam scores while controlling for student prior 
achievement levels: by year (before and after taking the APAI I PD), by student race/ethnicity, 
and by any interactions between year and race/ethnicity. Because students included in the file 
were in different grade levels in high school (some AP Exam takers were sophomores, some 
juniors, and some seniors), prior achievement on the state assessment was identified simply 
as assessment scores in the year prior to AP In other words, if a student's AP Exam took 
place in his or her senior year, the junior year state assessment was used to control for prior 
achievement. 

Results 

Descriptives 

Descriptive statistics were calculated to examine students' AP participation and performance 
and prior achievement levels across years, by race/ethnicity, by gender, and by AP subject. 
Results from these analyses are presented in Tables 2-7, with the difference between years 
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depicted as the delta (A) in each table. It should be noted that while the differences between 
years are discussed as being higher or lower, none of the differences were statistically 
significant at the p < .05 level. 

Table 2 presents the number of AP Exams taken across racial/ethnic groups and by gender, 
both before and after the teachers participated in the APAI I PD. The largest increase in AP 
Exams came from Hispanic students (an increase of 61 students), followed by an increase 
in exams by Asian American students (13 more exams in 2009-10); the number of AP Exams 
taken by African American, Native American, and white students stayed about the same. 
While females took more exams overall than males, they also increased more in the number 
of exams taken from 2008-09 to 2009-10 (increasing 43 exams compared to 35 exams for 
males). 


Table 2. 

Number of AP Exams Taken, by Students' Race/Ethnicity and Gender 



2008-09 

2009-10 

A 

Asian American 

168 

181 

13 

African American 

26 

30 

4 

Hispanic 

192 

253 

61 

Native American 

2 

3 

1 

White 

135 

134 

-1 

Female 

326 

369 

43 

Male 

197 

232 

35 


Table 3 presents the number of AP Exams taken by year for the subject areas that were 
the focus of this study. U.S. History was the most commonly taken exam for this group, 
both before and after the APAI I PD, while English Literature and Composition was the 
least common exam in both years. The exam with the largest increase after APAI I was U.S. 
Government and Politics (with an increase of 51 exams after the PD), followed by U.S. History 
(with an increase of 21 exams after the PD), and English Language and Composition (with an 
increase of 18 exams after the PD). Both English Literature and Composition and European 
History showed slight declines in exam taking (-3 and -9 exams, respectively). 


Table 3. 

Number of AP Exams Taken, by AP Subject 

2008-09 

2009-10 

A 

English Language and Composition 

137 

155 

18 

English Literature and Composition 

27 

24 

-3 

European History 

64 

55 

-9 

U.S. Government and Politics 

131 

182 

51 

U.S. History 

164 

185 

21 

Total 

523 

601 

78 


Table 4 presents students' scale scores on the ELA portion of their state assessment in the 
year prior to taking an AP course with their APAI I teacher. On average, ELA prior achievement 
was lower for students in AP courses taught after the APAI I PD (-4.16 points lower). 
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The study did not examine individual course taking prior to the AP course, but the state 
assessment scores suggest that this cohort of students may have been less well prepared 
than others to succeed in a rigorous AP course. While African American AP students in the 
2009-10 school year had higher prior achievement than African American AP students in the 
2008-09 school year, all other subgroups had lower prior achievement, including the Hispanic 
subgroup, which showed the largest increase in participation in AP after the APAI I PD. 
Although these results highlight a salient trend in terms of preparation, the differences were 
not statistically significant. 


Table 4. 

Mean Prior ELA Achievement on the State Assessment 


2008-09 

2009-10 

A 

Asian American 

375.24 

371.66 

-3.58 

African American 

374.65 

388.67 

14.01 

Hispanic 

383.77 

377.74 

-6.03 

White 

408.59 

407.16 

-1.44 

Female 

385.25 

380.73 

-4.51 

Male 

390.11 

386.38 

-3.73 

Average 

387.08 

382.92 

-4.16 


Table 5 presents students' AP performance by racial/ethnic subgroups and by gender. As 
expected given their prior achievement on the California Standards Tests (CSTs), African 
American students had higher AP Exam scores in the 2009-10 school year compared to the 
2008-09 school year. All other groups showed slightly lower AP scores; the average AP score 
in 2009-10 was -0.11 points lower than in 2008-10, although none of these differences were 
significant. 


Table 5. 

Mean AP Exam Score 



2008-09 

2009-10 

A 

Asian American 

1.99 

1.96 

-0.03 

African American 

2.19 

2.50 

0.31 

Hispanic 

2.13 

1.96 

-0.17 

White 

2.89 

2.81 

-0.08 

Female 

2.17 

2.04 

-0.13 

Male 

2.46 

2.39 

-0.07 

Average 

2.28 

2.18 

-0.11 


Tables 6 and 7 present students' mean prior achievement scores and mean AP scores 
for students before and after the APAI I PD by AP subject area. Students taking AP 
courses in European History, English Language and Composition, and English Literature 
and Composition in 2009-10 had lower prior ELA achievement than students in 2008-09 
(-19.71, -4.35, and -9.45 points lower, respectively), while students taking AP courses in 
U.S. Government and Politics and U.S. History in 2009-10 had slightly higher prior ELA 
achievement than students in 2008-09 (3.21 and 2.95 points higher, respectively). All 
differences in scale scores were not significant, with the exception of the scores for European 
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History. Without controlling for prior achievement levels, mean AP scores were lower after the 
APAI I PD on all exams except U.S. History (see Table 7), with the largest decline in European 
History (-0.65 points). All differences in mean AP scores were not significant, with the 
exception of European History. 


Table 6. 

Mean Scale Scores 


2008-09 

2009-10 

A 

English Language and Composition 

385.59 

381.25 

- 4.35 

English Literature and Composition 

414.41 

404.96 

- 9.45 

European History 

428.75 

409.04 

- 19.71 

U.S. Government and Politics 

365.07 

368.28 

3.21 

U.S. History 

385.14 

388.09 

2.95 


Table 7. 

Mean AP Scores 


2008-09 

2009-10 

A 

English Language and Composition 

1.98 

1.89 

- 0.09 

English Literature and Composition 

2.52 

2.25 

- 0.27 

European History 

3.14 

2.49 

- 0.65 

U.S. Government and Politics 

2.37 

2.15 

- 0.22 

U.S. History 

2.10 

2.34 

0.24 


Statistical Model 

An analysis of covariance (ANCOVA) was conducted on the APAI I sample to determine 
whether AP mean scores differed between the "before" and "after" APAI I groups, while 
controlling for prior achievement on the state ELA assessment. Table 8 presents the results 
of the ANCOVA after controlling for prior achievement. It indicates that the main effect for 
year (pre- versus post-APAI I) was not significant, while the main effect for race/ethnicity was 
statistically significant. 


Table 8. 

ANCOVA Results 


Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > f 

State Scale Score 

1 

506.53 

506.53 

577.68 

<.0001 

Year 

1 

0.58 

0.58 

0.66 

0.42 

Race/Ethnicity 

4 

30.19 

7.55 

8.61 

<.0001 

Year by Race/Ethnicity 

4 

1.49 

0.37 

0.42 

0.79 


A closer examination of the main effect for race/ethnicity utilizing the Scheffe post hoc 
procedure indicated that Asian American students scored significantly lower than African 
American and white students on their AP Exams, and that Hispanic students scored 
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significantly lower than African American students. No other differences were significant. The 
year by race/ethnicity interaction effect was not significant. 


Conclusion 

After APAI I, more Hispanic students of lower prior achievement (though not statistically 
significant) were enrolled in the AP courses of the APAI I teachers than before the APAI I PD. 
Despite this increase, there were no statistically significant differences between the students' 
AP scores before and after the APAI I PD. This suggests that the APAI I teachers were, in fact, 
able to provide rigorous AP instruction to a more diverse set of learners. 

There are two limitations to the current study that should be noted and considered when 
interpreting the findings. First, it is unclear whether increased Hispanic AP enrollments were 
caused by APAI I because no information exists about how students were admitted to AP 
courses. Second, the findings allow only for limited generalizability because the effects were 
examined within only one school district. Thus, it is not certain that similar results would be 
found were APAI I to be implemented in other districts or schools. 
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Appendix: APAI I Objectives (Summer 2009) 

Teachers 

What Does AP Learning Look Like? 

• Establish common definitions of rigor, differentiation, and scaffolding 

• Determine the differences and connections among rigor, differentiation, and scaffolding 

• Demonstrate and practice questioning techniques to increase understanding of texts 

• Model scaffolding in a variety of forms 

Data Analysis 

• Review College Board data and reports that can be used to inform AP learning 

• Explain strengths, deficiencies, and gaps with own curriculum, based on 
College Board data 

• Identify skills and topics from data reports that are needed to revise and enhance 
own curriculum 

• Assess the qualities of a good reader and a good writer 

• Evaluate AP free-response samples 

• Develop exercises that scaffold AP learning 

Teaching Reading 

• Identify successful strategies for teaching students to read AP-level texts 

• Develop strategies to teach a heterogeneous AP group to read AP-level texts 

• Review own curriculum to determine whether/how current purpose, strategies, and 
assessments are related 

• Connect purpose, strategies, and assessments of assignments to AP syllabus and 
unit plans 

• Distinguish appropriate reading strategies for the selected text 

Teaching Writing 

• Identify successful strategies for teaching students to write AP-level assignments 

• Construct a writing assignment and develop teaching strategies for the assignment 

• Review own curriculum to determine whether/how current purpose, strategies, and 
assessments are related 

• Estimate the proportion of types of writing assignments given in class 

• Compare and contrast content of AP Exam with student expectations in AP course 
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Administrators 

What Does AP Learning Look Like? 

Curricular and Instructional Leadership 

• Analyze AP performance as a function of departmental success as opposed to individual 
teacher success 

• Analyze course sequencing and scheduling practices to support AP student achievement 

• Identify gateway AP courses to introduce students to rigorous AP courses 

• Analyze classroom observation processes to effectively serve as an instructional leader in 
the AP classroom 

• Determine how to assist teachers in differentiation of instruction in the AP classroom 

• Identify gateway academic skills students should possess prior to entering the 
AP classroom 

• Develop a plan of preparation for first-time, nontraditional AP students 

• Analyze teacher preparation in order to effectively teach an AP class 

• Identify traits and characteristics of quality AP teacher-selection decisions 

Data Analysis and Community Leadership 

• Identify how to use the AP Course Description, the AP Exam, AP score reports, and the 
PSAT/NMSQT® as tools to inform instruction 

• Define the steps for ordering materials and reports, as well as for completing the 
AP Course Audit process 

• Develop a plan to communicate the AP Program to parents and community members 
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department actively supports the 
College Board’s mission by: 


Providing data-based solutions to important educational problems and questions 

Applying scientific procedures and research to inform our work 
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Publishing findings and presenting our work at key scientific and education conferences 
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